1. Field of the Invention
The present invention relates to a stereo camera apparatus for distance measurement and more specifically, to a stereo camera apparatus for distance measurement capable of correcting a shift generated between a left image sensor and a right image sensor.
2. Description of the Related Art
With improvement of image processing techniques, cameras with high performance have become relatively low priced. Therefore, cameras are applied to many areas.
For example, conventionally, in order to aid a driver of an automobile not to crash or to control the space between one's own automobile and an obstacle in front, a millimeter wave radar is mounted on the automobile to detect the obstacle in front for alerting a driver or controlling the automobile.
There has been an attempt to actualize this technique with a camera. When actualizing the technique with a camera, a stereo camera is mounted on an automobile to obtain a parallax image of an obstacle in front. Further, an image processing unit calculates the distance to the obstacle based on the parallax image obtained by the stereo camera. After the distance is determined, the same aid for a driver is possible.
FIG. 1 is a view for explaining a principle of distance measurement by a stereo camera in which a left camera C0 (left image sensor) and a right camera C1 (right image sensor) are positioned parallel to each other.
The left camera C0 and the right camera C1 are positioned at a distance B from each other. The focal lengths, the optical centers, and the image surfaces for the left camera C0 and the right camera C1 are as follows.
(For the Left Camera C0)
The focal length is f, the optical center is O0 and the image surface is s0.
(For the Right Camera C1)
The focal length is f, the optical center is O1 and the image surface is s1.
An image of an object “A” that is positioned at a distance “d” in the optical axis direction from the optical center O0 of the camera C0 is focused on a point P0 which is a node of a line A-O0 and the image surface s0.
On the other hand, for the right camera C1, an image of the same object “A” is focused on a point P1 on the image surface s1. Here, a node of a line that passes through the optical center O1 of the right camera C1 and is parallel to the line A-O0, and the image surface s1 is assumed as a point P0′, and the distance between the point P0′ and the point P1 is assumed as a distance “p”.
The position of the point P0′ on the image surface s1 corresponds to the position of the point P0 on the image surface s0 of the camera C0, so that the distance “p” expresses a shift amount of positions of the same object in the images taken by two different cameras (the left camera C0 and the right camera C1) and the distance “p” is so called a “parallax”.
In this case, as a triangle defined by A-O0-O1 and a triangle defined by O1-P0′-P1 are orthomorphic, the following equation is obtained.d=Bf/p 
When the distance B (base line length) between the left camera C0 and the right camera C1, and the focal length “f” are known, the distance “d” can be obtained from the parallax “p”.
As described above, for the distance measurement by the stereo camera, principally, the optical axes of two cameras are presumed to be parallel to each other. However, for the actual stereo camera, it is difficult to have the optical axes of the two cameras strictly parallel because of structural tolerance, variation with time by temperature or vibration, or the like. Therefore, a technique to correct images is important and a technique to electronically correct a shift in images taken by the stereo camera is disclosed in Japanese Laid-open Patent Publication No. 11-325890, for example.
Japanese Laid-open Patent Publication No. 11-325890 discloses an image correction apparatus in which an initial position of a base marker (for example, a specific position at a front edge of a hood of a car) within a field of vision of a stereo camera is stored, and then one of the images is shifted in the upward, downward, leftward, or rightward to correct the shift of the base marker in the images taken for the distance measurement.
Here, for obtaining an image, a rolling shutter (or line scan) in which pixels in each scanning line of a frame of an image are sequentially exposed, and a global shutter in which all the pixels in a frame of an image are exposed at the same time, are proposed. Many cameras including a Complementary Metal Oxide Semiconductor (CMOS) type image sensor adopt the rolling shutter to reduce electrical power consumption.
FIG. 2 is a view for explaining an example of operational timing of a rolling shutter.
A horizontal axis expresses time where the time passes from the left-side to the right-side. An “R” (row) and a “C” (column) express a coordinate position of each pixel. As shown in FIG. 2, the CMOS image sensor adopting the rolling shutter sequentially reads out pixel values of pixels at each of the columns in each of the rows. Concretely, the CMOS image sensor first exposes a pixel at the first column in the first row (R1C1) to output its pixel value, exposes a next pixel at the second column in the first row (R1C2) to output its pixel value, and then sequentially exposes each of the pixels in the first row to output their pixel values. When the pixel values of all of the pixels in the first row are output, the CMOS image sensor sequentially exposes each of the pixels in the second row to output their pixel values.
Therefore, there is a time difference in exposing (or obtaining) the pixel values for the pixels. For example, when the number of the pixels in the first row is “M”, the time difference in exposing the pixel values for the pixels at the same column in the first row and the second row becomes a product of “pixel clock” and “M”.
Because of the time difference, for the camera adopting the rolling shutter, when taking an image of an object moving fast in the horizontal direction, the object is distorted in the oblique direction in the image (hereinafter, this phenomenon is referred to as a distortion of a moving object).
FIG. 3 is a view for explaining an example of a distortion generated by the rolling shutter. FIG. 3 shows an object 50 having a rectangular shape and moving rightward as shown by the arrow, and an image 52 of the object 50 taken by a camera adopting the rolling shutter.
As described above, the pixel values of the pixels in the lower rows are obtained later. Therefore, the positions of the pixels at the lower rows are shifted rightward as the object 50 is moving rightward to distort the object 50 to be shown like a parallelogram 50a in the image 52.
The distortion of a moving object may be corrected by image processing. However, according to the method disclosed in Japanese Laid-open Patent Publication No. 11-325890, it is difficult to correct the distortion of a moving object.
FIG. 4A and FIG. 4B are views for explaining an example of a problem in a correction of the distortion of a moving object.
FIG. 4A shows a left-side image 54 and a right-side image 56 respectively obtained by the left camera C0 and the right camera C1 (see FIG. 1) of a stereo camera, before a correction.
In this case, it is assumed that the right camera C1 that obtained the right-side image 56 is positioned to face in a bit upper direction relative to the left camera C0 that obtained the left-side image 54. Therefore, the object 55a (or 55b) which is common for the left-side image 54 and the right-side image 56 is positioned “a” rows lower in the right-side image 56 than in the left-side image 54.
The object 55a, which is not a moving object, has a rectangular shape. As for the object 55a, distortion of a moving object is not generated.
FIG. 4B shows the left-side image 54 and an example of a corrected image 56a of the right-side image 56 obtained by correcting the right-side image 56 in accordance with a method such as disclosed in Japanese Laid-open Patent Publication No. 11-325890. Here, by the correction, the right-side image 56 obtained by the right camera C1 is vertically shifted upward for “a” rows so that the objects in the left-side image 54 and the corrected image 56a are positioned at the same height.
However, such a correction is performed after all the pixels in a frame of the images are obtained, so that the time difference is not corrected. Therefore, the pixels in the first row of the left-side image 54 obtained by the left camera C0 corresponds to the pixels in the (a+1)th row of the corrected image 56a of the right-side image 56 obtained by the right camera C1. It means that, when assuming that the left camera C0 and the right camera C1 start obtaining the left-side image 54 and the right-side image 56 at the same time, there is a time difference corresponding to “a” rows between the pixels in the same row (at the same height) of the left-side image 54 and the corrected image 56a. 
Thus, if the object is a moving object like the object 55b shown by dotted lines in FIG. 4A, where the positions of the pixels at the lower rows are shifted rightward, a difference other than the parallax is generated between the objects 55b in the left-side image 54 and the right-side image 56 in the horizontal direction. This difference is not corrected even after the correction by the method as disclosed in Japanese Laid-open Patent Publication No. 11-325890 is performed, as shown in FIG. 4B.
Thus, according to the above method, if images of an object moving fast in the horizontal direction are obtained, because of the distortion of a moving object described above with reference to FIG. 3, the detected parallax even after using the method disclosed in Japanese Laid-open Patent Publication No. 11-325890 is performed, is not accurate and therefore, the distance calculated based on the parallax is not accurate either.